Skip to content

feat(evaluation): independent retrieved-learning sampling; retire retrieval-log capture#334

Merged
guangyu-reflexio merged 3 commits into
mainfrom
feat/retrieved-learning-coverage
Jul 15, 2026
Merged

feat(evaluation): independent retrieved-learning sampling; retire retrieval-log capture#334
guangyu-reflexio merged 3 commits into
mainfrom
feat/retrieved-learning-coverage

Conversation

@yyiilluu

@yyiilluu yyiilluu commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Paired with the enterprise PR that rewires the offline playbook tuner onto a single per-playbook impact signal. Merge this first, then the enterprise repo pins it.

Why

The tuner now reads the per-playbook impact verdict in retrieved_learning_evaluation and nothing else. Two consequences land in OSS:

  1. The retrieval-capture subsystem becomes unreferenced — it existed to record what was served, which Interaction.retrieved_learnings now carries.
  2. The tuner needs two things OSS didn't offer: coverage it can buy without side effects, and a windowed read that doesn't silently truncate.

What's here

Independent per-family sampling

New AgentSuccessConfig.retrieved_learning_sampling_rate, default None = inherit sampling_rate — an org that never opts in keeps exactly its previous behavior and cost.

New agent_success_evaluation/sampling.py: the publish gate samples both families once, schedules when either admits the session, and passes two booleans to the runner. One computation site, so the families cannot diverge. A session sampled only for retrieved-learning never pays the session-success judge — which is the point: it lets an operator buy coverage without also raising the agent-success bill.

Direct callers (regen jobs, on-demand grade) run both families as before. Sampling is a scheduling decision, not a runner one.

degraded is settled, not broken

failed/pending committed nothing and had no retry trigger, so such a session was invisible forever. The publish callback now re-arms it on the existing scheduler, bounded to 3 attempts.

degraded is deliberately NOT retried. It is an applied, fingerprint-fenced commit: the rows are persisted and only the failed chunks carry NULL impact. Retrying it re-executes every relevance+impact chunk and delete/re-inserts already-committed rows — and a deterministically degrading chunk (over-length learning, content-filter refusal) degrades again on every attempt, so a bounded sweep burned 4× the judge bill and bought nothing.

Instead: new SETTLED_RETRIEVED_STATUSES (= TERMINAL | {degraded}), exposed via an optional statuses arg on the terminal-state fence, so consumers read those rows directly. Contract-tested that widening the set does not weaken the freshness check — a degraded session whose transcript changed is still rejected.

⚠️ Best-effort retry: the attempt counter and schedule are in-process (GroupEvaluationScheduler is an in-memory heap), so a deploy or task recycle inside the delay window drops pending retries, and with N tasks a session can start N independent chains. Durability needs a persisted queue — not done here.

Exhaustive windowed read

get_retrieved_learning_evaluation_results_in_window() on the base + SQLite. The existing paged reader hard-caps at limit=100 with no cursor; a consumer that treats the window as the complete set must not inherit that silent truncation. Contract-tested at 0 / 100 / 101 / 250 rows.

Retrieval-log retirement

Deletes PlaybookRetrievalLog / PlaybookRetrievalLogItem, RetrievalLogMixin, and the configure_retrieval_capture_hook search hook.

Deliberately KEPT:

  • the AuditEntityType "playbook_retrieval_log" literal — historical audit_events rows still carry it, and removing it makes Pydantic reject them on read;
  • the retention targets + cascade — the tables still hold PII until a later release drops them. Removing a purge before dropping its table strands PII.

Verification

  • 4613 passed; the 10 failures are pre-existing sqlite-vec/FTS ordering flakes, confirmed by running the same suite on clean main.
  • Ruff + Pyright clean.
  • The sampling-default test now constructs the config without the field. The previous version passed None explicitly, so it asserted its own argument — which is exactly how a 0.1 default drifted in unnoticed. Verified the new test goes red if the drift is reintroduced.

Summary by CodeRabbit

  • New Features
    • Added independent, deterministic sampling for agent-success vs retrieved-learning evaluations, with a new optional retrieved_learning_sampling_rate config (inherits existing sampling when unset).
    • Added inclusive time-window reads for retrieved-learning evaluation results and bounded retry scheduling for pending/failed retrieved-learning outcomes.
  • Bug Fixes
    • Preserved read compatibility for existing audit records containing legacy request audit entity types.
  • Documentation
    • Clarified retained behavior for playbook_retrieval_logs during the retirement period.
  • Refactor
    • Removed retrieval-log models and retrieval-capture hook APIs.
  • Tests
    • Expanded per-family sampling, window-read, and retry coverage; removed obsolete retrieval-log and unified-search capture-hook tests.

@coderabbitai

coderabbitai Bot commented Jul 11, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: d5b44ec5-4f21-488a-8b8d-2ac59dcf75de

📥 Commits

Reviewing files that changed from the base of the PR and between f7f3758 and a34c3e5.

📒 Files selected for processing (20)
  • reflexio/models/api_schema/domain/entities.py
  • reflexio/models/api_schema/domain/governance.py
  • reflexio/models/config_schema.py
  • reflexio/server/extensions.py
  • reflexio/server/services/agent_success_evaluation/runner.py
  • reflexio/server/services/agent_success_evaluation/sampling.py
  • reflexio/server/services/generation_service.py
  • reflexio/server/services/storage/retention.py
  • reflexio/server/services/storage/sqlite_storage/playbook/_eval_results.py
  • reflexio/server/services/storage/storage_base/__init__.py
  • reflexio/server/services/storage/storage_base/_retrieval_log.py
  • reflexio/server/services/storage/storage_base/playbook/_eval_results.py
  • reflexio/server/services/storage/storage_base/retrieved_learning_state.py
  • reflexio/server/services/unified_search_service.py
  • tests/server/services/agent_success_evaluation/test_sampling.py
  • tests/server/services/storage/test_storage_contract_retrieval_logs.py
  • tests/server/services/storage/test_storage_contract_retrieved_learning_evals.py
  • tests/server/services/test_generation_service_scheduling.py
  • tests/server/services/test_unified_search_service.py
  • tests/server/services/test_unified_search_session_dedup.py
💤 Files with no reviewable changes (6)
  • tests/server/services/test_unified_search_session_dedup.py
  • reflexio/server/services/storage/storage_base/_retrieval_log.py
  • tests/server/services/storage/test_storage_contract_retrieval_logs.py
  • reflexio/server/services/storage/storage_base/init.py
  • reflexio/server/services/unified_search_service.py
  • tests/server/services/test_unified_search_service.py
🚧 Files skipped from review as they are similar to previous changes (8)
  • reflexio/server/services/agent_success_evaluation/sampling.py
  • reflexio/server/services/storage/retention.py
  • reflexio/models/api_schema/domain/entities.py
  • reflexio/models/api_schema/domain/governance.py
  • tests/server/services/agent_success_evaluation/test_sampling.py
  • reflexio/server/extensions.py
  • reflexio/server/services/agent_success_evaluation/runner.py
  • reflexio/server/services/generation_service.py

📝 Walkthrough

Walkthrough

This change removes retrieval-log capture APIs and models, adds independent agent-success and retrieved-learning sampling, introduces bounded retrieved-learning retries, and expands retrieved-learning storage interfaces with exhaustive window reads.

Changes

Retrieved-learning evaluation lifecycle

Layer / File(s) Summary
Retire retrieval-capture and retrieval-log APIs
reflexio/models/api_schema/domain/entities.py, reflexio/models/api_schema/domain/governance.py, reflexio/server/extensions.py, reflexio/server/services/unified_search_service.py, reflexio/server/services/storage/..., tests/server/services/test_unified_search*
Retrieval-log models, storage mixins, unified-search capture hooks, and related test wiring are removed; historical audit compatibility and retention behavior remain documented.
Add per-family sampling configuration
reflexio/models/config_schema.py, reflexio/server/services/agent_success_evaluation/sampling.py, tests/server/services/agent_success_evaluation/test_sampling.py
A validated retrieved-learning sampling rate and deterministic per-session sampling functions support independent evaluation families, with fallback to the existing sampling rate.
Schedule and execute sampled evaluations
reflexio/server/services/agent_success_evaluation/runner.py, reflexio/server/services/generation_service.py, tests/server/services/test_generation_service_scheduling.py
Scheduling passes family-specific flags to evaluation, and failed or pending retrieved-learning outcomes are retried up to three times without rerunning agent-success evaluation.
Extend retrieved-learning result storage
reflexio/server/services/storage/storage_base/playbook/_eval_results.py, reflexio/server/services/storage/storage_base/retrieved_learning_state.py, reflexio/server/services/storage/sqlite_storage/playbook/_eval_results.py, tests/server/services/storage/test_storage_contract_retrieved_learning_evals.py
Storage adds exhaustive inclusive time-window reads, optional agent-version filtering, deterministic ordering, and clarified persisted-status semantics.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GenerationService
  participant Sampling
  participant GroupEvaluationScheduler
  participant EvaluationRunner
  GenerationService->>Sampling: select evaluation-family flags
  Sampling-->>GenerationService: return agent-success and retrieved-learning flags
  GenerationService->>GroupEvaluationScheduler: enqueue sampled evaluation
  GroupEvaluationScheduler->>EvaluationRunner: run_group_evaluation(flags)
  EvaluationRunner-->>GroupEvaluationScheduler: return retrieved-learning status
  GroupEvaluationScheduler->>GroupEvaluationScheduler: retry failed or pending status up to 3 times
Loading

Possibly related PRs

Suggested reviewers: guangyu-reflexio

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 52.38% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly reflects the two main themes of the PR: independent retrieved-learning sampling and removal of retrieval-log capture.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/retrieved-learning-coverage

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
reflexio/server/services/agent_success_evaluation/runner.py (1)

110-121: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Stale step numbering in comments/docstring after inserting the admission gate.

The docstring's "Steps" list (lines 110-121) still enumerates the old 8-step flow and doesn't mention the new per-family admission step. Additionally, both the "already evaluated" check (Line 188) and "fetch interactions" (Line 218) are now labeled step "4", since inserting "3. Per-family admission" shifted subsequent steps without renumbering them.

✏️ Suggested renumbering
-    # 4. Check if agent success is already evaluated — skipped in
+    # 5. Check if agent success is already evaluated — skipped in
...
-    # 4. Fetch interactions for all requests
+    # 6. Fetch interactions for all requests

Also update the docstring "Steps" list to include the new per-family admission step.

Also applies to: 171-188, 218-218

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@reflexio/server/services/agent_success_evaluation/runner.py` around lines 110
- 121, Update the Steps docstring in the evaluation runner to include the
per-family admission step and renumber all subsequent steps through the final
operation-state update. Correct the inline step labels near the
already-evaluated check and interaction fetching so each label matches the
updated flow, without changing runtime behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@reflexio/server/services/storage/sqlite_storage/playbook/_eval_results.py`:
- Around line 417-419: The status selection in the affected retrieval method
currently treats an explicitly empty statuses override as unset. Update the
logic around the statuses parameter to use TERMINAL_RETRIEVED_STATUSES only when
statuses is None, while preserving an empty frozenset so it matches no persisted
state.

---

Nitpick comments:
In `@reflexio/server/services/agent_success_evaluation/runner.py`:
- Around line 110-121: Update the Steps docstring in the evaluation runner to
include the per-family admission step and renumber all subsequent steps through
the final operation-state update. Correct the inline step labels near the
already-evaluated check and interaction fetching so each label matches the
updated flow, without changing runtime behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: ec1a1635-fb53-4431-8518-04a4bb889aff

📥 Commits

Reviewing files that changed from the base of the PR and between c486b46 and 82c048b.

📒 Files selected for processing (20)
  • reflexio/models/api_schema/domain/entities.py
  • reflexio/models/api_schema/domain/governance.py
  • reflexio/models/config_schema.py
  • reflexio/server/extensions.py
  • reflexio/server/services/agent_success_evaluation/runner.py
  • reflexio/server/services/agent_success_evaluation/sampling.py
  • reflexio/server/services/generation_service.py
  • reflexio/server/services/storage/retention.py
  • reflexio/server/services/storage/sqlite_storage/playbook/_eval_results.py
  • reflexio/server/services/storage/storage_base/__init__.py
  • reflexio/server/services/storage/storage_base/_retrieval_log.py
  • reflexio/server/services/storage/storage_base/playbook/_eval_results.py
  • reflexio/server/services/storage/storage_base/retrieved_learning_state.py
  • reflexio/server/services/unified_search_service.py
  • tests/server/services/agent_success_evaluation/test_sampling.py
  • tests/server/services/storage/test_storage_contract_retrieval_logs.py
  • tests/server/services/storage/test_storage_contract_retrieved_learning_evals.py
  • tests/server/services/test_generation_service_scheduling.py
  • tests/server/services/test_unified_search_service.py
  • tests/server/services/test_unified_search_session_dedup.py
💤 Files with no reviewable changes (7)
  • reflexio/server/services/storage/storage_base/_retrieval_log.py
  • tests/server/services/storage/test_storage_contract_retrieval_logs.py
  • tests/server/services/test_unified_search_session_dedup.py
  • reflexio/server/services/storage/storage_base/init.py
  • reflexio/models/api_schema/domain/entities.py
  • tests/server/services/test_unified_search_service.py
  • reflexio/server/services/unified_search_service.py

Comment thread reflexio/server/services/storage/sqlite_storage/playbook/_eval_results.py Outdated
…rieval-log capture

Reconciled onto OSS main (per-interaction RLE, #348): keeps main's
evaluation_version freshness check + the new statuses-param terminal-state
match. Independent retrieved_learning_sampling_rate + bounded retry sweep +
retrieval-log retirement (AuditEntityType literal kept for read-compat).
SETTLED/statuses machinery retained here; dropped in a follow-up per the
reconciliation design.
The retrieved-learning-evals contract helper is now
_result(storage, kind, learning_id), but the window-read tests added
by #334 still called it with 2 args, raising TypeError at collection.
Update every 2-arg call site to pass the backend handle.
…NAL-only

The offline tuner consumes TERMINAL retrieved-learning state only. main already
retries degraded -> complete, so a degraded session self-heals on the next
scheduled run and is picked up once it settles. This reverts the #334 SETTLED
surface to main's param-less form:

- Remove SETTLED_RETRIEVED_STATUSES from retrieved_learning_state.
- Remove the statuses= param from get_matching_retrieved_learning_terminal_state
  on both the abstract base and SQLite backend; match TERMINAL_RETRIEVED_STATUSES
  directly (preserving the evaluation_version + fingerprint fence).
- Drop the two SETTLED-specific storage contract tests; refresh dangling
  SETTLED_RETRIEVED_STATUSES comments in generation_service and its scheduling
  test (degraded is re-judged fresh on the next run, not read directly).
@guangyu-reflexio guangyu-reflexio force-pushed the feat/retrieved-learning-coverage branch from f7f3758 to a34c3e5 Compare July 15, 2026 11:40
@guangyu-reflexio guangyu-reflexio merged commit 966689e into main Jul 15, 2026
1 check passed
@guangyu-reflexio guangyu-reflexio deleted the feat/retrieved-learning-coverage branch July 15, 2026 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants